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Abstract. This paper argues for the use of cognitive modeling to gain a detailed and dynamic look into user 
experience during game play. Applying cognitive models to game play data can help researchers 
understand a player’s attentional focus, memory status, learning state, and decision strategies (among other 
things) as these cognitive processes occurred throughout game play. This is a stark contrast to the common 
approach of trying to assess the long-term impact of games on cognitive functioning after game play has 
ended. We describe what cognitive models are, what they can be used for and how game researchers could 
benefit by adopting these methods. We also provide details of a single model - based on decision field 
theory - that has been successfully applied to data sets from memory, perception, and decision making 
experiments, and has recently found application in real world scenarios. We examine possibilities for 


applying this model to game-play data. 

1. INTRODUCTION 

A major goal of video game research is to 
understand and influence what a player is thinking 
during game play, and perhaps to effect long term 
changes in the game player. One major theme in 
this research includes game impact on personality 
traits and emotional states - most famously, effects 
on player aggressiveness. This type of research is 
typically carried out via survey methods. Players 
answer a battery of questions before the game 
begins to assess their personal traits and current 
emotional state. After game play has completed, 
the player answers more questions and any 
changes are attributed to the effects of the 
intervening game play period. Another prominent 
theme is the effect of games on some aspect of 
cognition (e.g., spatial skills [5]). Although this 
second research theme is concerned with cognitive 
effects of games, it is often carried out by pretesting 
the participant using a standard laboratory task 
(e.g., a speeded search task) to assess the trait of 
interest (e.g., visual acuity), followed by a game 
play period, and then posttesting using the same 
standard laboratory task again to see if 
performance has changed. 

These approaches take a bird’s eye view of the 
cognitive phenomena underlying game play. They 
treat the mind like an impenetrable black box, 
observing or manipulating inputs to the cognitive 
system, and observing the concomitant outcomes. 
Although these research endeavors are valuable, 
they take an indirect route to understanding 
cognition during play. In both approaches outlined 
above, game play effects are measured after the 
fact. In the case of questionnaire methods, the data 
is subjective - participants give some indication of 


the game’s effects through their answers to various 
questions. And although the second research 
approach aims at understanding something about 
cognition as it pertains to games, it too focuses on 
effects and measures indirectly by assessing 
changes after the fact. 

What is needed is an approach that allows one to 
track cognitive effects of games during the game 
play session. As the player progresses through the 
game, seeking to accomplish various goals, making 
decisions, all manner of cognitive phenomena come 
to bear. Learning is required (e.g., what strategies 
worked before?). Attention has to be allocated. 
Memories of previous outcomes have to be 
accessed. Decisions have to be made. The 
outcome of these cognitive events translate into the 
player’s observable performance in the game, their 
level of enjoyment or accomplishment, their 
learning. Understanding these dynamic events as 
they unfold throughout the course of play, rather 
than trying to infer something about them 
subjectively or indirectly after the game is over, 
would be of great value to the designer of games 
concerned with changing behavior, communicating 
messages, or just maximizing engagement and fun. 
In attempting to maximize game efficacy, the 
designer would likely benefit from knowing what a 
player is looking at when making decisions, what 
dimensions are most salient, which dimensions are 
routinely ignored, and when options are confusable. 

In recent years, there have been efforts to peer into 
the inner workings of the mind during the game play 
events that cause them. Brain imaging techniques 
(e.g., fMRI) have been used to associate brain 
activity known to occur during aggressive thought 
with violent game content [8], Other studies have 
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tracked psychophysiological events (e.g., EEG) to 
infer mental states during play [1]. This approach to 
studying player cognition during game play is a 
welcome addition to the field. This research gives 
us valuable data synched in time to game events, 
and we can learn a lot by trying to interpret it. 
However, at this point in time, both imaging and 
psychophysiolical data are difficult and expensive to 
obtain, the measures are still relatively crude, and 
findings are often difficult to interpret. The degree 
to which one can relate the observed bodily states 
to aspects of cognition such as attention or 
executive function is an issue of lively (sometimes 
withering) debate [7], 

Luckily, there is another approach to understanding 
cognitive phenomena at our disposal - cognitive 
modeling. For the past several decades, cognitive 
psychologists have devised and tested scores of 
detailed mathematical models that offer precise 
accounts of the cognitive underpinnings of 
behavior, and demonstrated their links to theoretical 
structures like memory and attention. Given the 
mature state of this field, as well as its widespread 
representation throughout academia, it is surprising 
to find that it has very little representation in the 
game studies literature (although some applications 
are noted below). The objective of this paper is to 
provide a basic understanding of what cognitive 
models can provide researchers, and to advocate 
their use in studying video games. 

2. COGNITIVE MODELING 

What, exactly, is a “cognitive model"? A cognitive 
model is a mathematical interpretation (i.e., 
specification) of the set of principles embodied in a 
theory of cognition. Cognitive models make specific 
assumptions about the information represented in 
the cognitive system (e.g., words and their 
meanings), along with the processes acting on this 
information to produce observable cognitive 
behavior (e.g., classifying an object). More 
concretely, a model receives inputs like a person in 
an experiment (e.g., size of objects on a screen), 
performs mental operations (e.g., like comparing 
perceived stimulus information to information stored 
in memory), and outputs a response (e.g., emits a 
classification of the object). 

Models such as these are valuable for several 
reasons. First, they require a researcher to move 
past the initial stages of theorizing - often 
characterized by vague verbal descriptions of 
mental entities and their interactions - to taking a 
detailed, specific stance on these quantities and 
relationships. Doing this affords the research 


community a better opportunity to evaluate and 
criticize a theory’s quality. Second, making detailed 
quantitative statements in a cognitive model allows 
a researcher to make precise, testable predictions. 
A third benefit is that simulating model behavior on 
a computer can lead to unexpected observations 
and insights that the researcher might not otherwise 
have reached. It is widely agreed in the modeling 
community that this is an important benefit of 
modeling. 

There currently exists a wide array of cognitive 
models that have been vetted over the years by 
many experiments and data sets. These models 
elucidate a range of topics. Many models are 
designed to capture steady-state performance in 
cognitive tasks like recognition memory, 
discrimination ability, attention allocation, to name a 
few. These models are intended to account for 
specific, circumscribed aspects of cognition such as 
recognition, categorization, attention, etc. Another 
class of models - known as connectionist models 
(also called neural net or parallel distributed 
processing models) - mimic fundamental aspects of 
brain anatomy (i.e., populations of single 
processing units or artificial neurons communicating 
activation levels back and forth) and capture 
learning over the course of many training trials. A 
third class of models - known as cognitive 
architectures (e.g, ACT-R, EPIC, Soar) - attempt to 
capture several aspects of cognition in a single 
unified framework (e.g., attentional processes, 
memory, visual search tendencies), reflecting the 
fact that all these processes come into play 
simultaneously in the human cognitive system. 
Cognitive architectures have found wide application 
in human-computer interaction research and have 
even made their way into game research to some 
extent [4], Existing applications of cognitive 
modeling in game research tends to take on a 
computer-science flavor. These models are 
valuable tools for making the game respond to the 
player in interesting ways or to create “smarter” 
non-player characters [3], Our aim in this paper is 
to encourage much more widespread adoption of 
these techniques for gaining general understanding 
of the cognitive capacities invoked during video 
game play. 

3. APPLYING COGNITIVE MODELS TO GAME 
STUDIES 

The value of models for game research lies in the 
fact that models require inputs and produce 
outputs. In between they offer precise statements 
about attention, learning, decision strategies and 
biases, and so on. In doing so, a model often tells 
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the researcher why performance looks as it does. 
Although a model can’t tell the designer exactly how 
to craft a game environment that teaches or 
entertains, discovering that current inputs place 
unrealistic demands on attention might offer 
guidance by narrowing the range of necessary 
modifications to gain desired results. An important 
detail, of course, is how one goes about applying 
these models. 

Within a cognitive model lie parameters that capture 
the modeled quantities (e.g., attention weights, 
learning rate, response biases). These values are 
indicators of the mental underpinnings of 
observable behavior. In order to make inferences 
about cognition, these models are often “fit” to a set 
of data. The computer takes the output of the 
model (i.e., predicted responses to events), 
compares it to player data (i.e., actual responses) 
and adjusts the internal parameter values (i.e., 
changes assumptions about attention, etc.) until the 
predicted responses are as close as possible to the 
data. The resulting adjusted parameter values 
indicate things like how confusable the stimuli were 
or which stimulus dimensions garnered the most 
attention. These parameter values can be used to 
make predictions for the player in later game 
sessions or scenarios. 

It is also important to verify that what the model tells 
us is correct. In order to do this, researchers often 
attempt to fit a model to data using fixed parameter 
values gleaned from prior knowledge of the 
research participant. Achieving a good model fit 
(i.e., a good prediction of player performance) by 
setting parameter values a priori is a powerful 
demonstration that one understands the player’s 
cognitive processes during play. 

One way to obtain fixed parameter values for a 
priori prediction is to fit the model (by adjusting free 
parameters) to one data set, and then use the best- 
fitting parameter values to see if the model 
accounts for additional data sets (without re- 
adjusting the parameter values). Another way to 
demonstrate our understanding is to set the model 
parameters based on something else we already 
know about the player. 

For example, one could take advantage of the kinds 
of data acquired through the survey methods 
described above. One recent (non-game) study 
used results from a survey designed to assess 
whether a person has an “action” orientation 
(tendency to accept risks to expedite achieving a 
goal) or a “state” orientation (tendency to be more 
deliberative in order to avoid risks). Scores on this 
questionnaire were converted into parameter values 


in a cognitive model and used to predict response 
probabilities and response time distributions in a 
sports-related task [6], Such an approach grounds 
model parameters in knowledge about the 
participant even before experimental manipulation 
begins, and can still enable the model to make 
interesting predictions about behavior. 

Another possibility would be for the researcher to 
set model parameters to reflect instructions given to 
the player (either before the game or inside the 
game). Instructing a player to pay attention only to 
RED enemies, for example, should be reflected in a 
model’s attention weight parameters (assuming the 
model has them) and consequently in the model's 
predicted response probabilities (and hopefully lead 
to a good model fit). An important long-term goal of 
modeling is to find parameter values that can lead 
to valid predictions across several experimental 
conditions without the need to adjust parameters to 
account for each data set. 

One challenge to applying cognitive models to data 
from video games is that events of interest must be 
operationally defined. For example, some 
agreement might need to be reached about what 
constitutes “fighting or fleeing” in a game scenario. 
Another example would be determining what 
quantifies as a response option. Depending on the 
question under study, it may be wise to compare 
performance only in situations with a constant 
number of response options. Such apples-to- 
apples comparisions might be necessary when 
trying to determine response probabilities or 
response time distributions. 

4. DECISION FIELD THEORY 

Among the many aspects of cognition that can be 
modeled and examined in games, perhaps the most 
natural starting point is to look at decision making. 
One popular class of models that illuminates 
decision making is known as “sequential sampling” 
models. Sequential sampling models simulate the 
accumulation of information (i.e. sampling) over 
time in support of each choice alternative, leading 
to the eventual selection of one option over others. 
Decisions are triggered by internal choice 
thresholds - the first accumulation process to reach 
threshold wins, and the corresponding choice is 
made. Figure 1 depicts this sampling process for 
three choice options. 


231 




Figure 1: Information accumulation for three choice 
options 

In this section, we describe a sequential sampling 
model based on Decision Field Theory [2], 
Variants of this model have been successfully 
applied to a wide range of phenomena, including 
decision making, perception, and memory, among 
others. The model has mostly been applied to data 
from standard laboratory tasks, but has recently 
been used to explain decision making in a sports 
judgment task [6]. 

On a given experimental trial (in the context of 
games, an operationally defined recurring event), 
the model assumes that each set of choice options 
can be characterized by values along salient 
dimensions. For example, when trying to choose 
the best weapon for a fight, the player might 
consider three weapons along dimensions such as 
strength, range, and ammunition supply. Each 
weapon has its own set of values on these 
dimensions, and the player makes some 
assessment of these values. Table 1 illustrates 
some hypothetical values. 

Table 1: Hypothetical dimension values for weapon 

choices 



Strength 

Range 

Ammo 

Option 1 

1.0 

50 

0.6 

Option 2 

0.5 

150 

0.8 

Option 3 

0.7 

100 

0.8 


Also, each player is likely to display some 
difference in preference for the choice dimensions. 
For example, the player's decision might be most 
strongly influenced by the strength dimension 70% 


of the time. Range might be the most influential 
dimension 20% of the time, and ammo only 10% of 
the time. These values are model parameters. The 
model uses these values on each trial, along with 
other parameters representing initial biases, 
memory from trial to trial, and similarity between 
options, to produce a decision. 

Figure 1 displays a characteristic example. The 
figure shows the (simulated) stochastic 
accumulation over time of evidence (to the cognitive 
system) in favor each of the three options. The first 
option to reach an internal decision threshold “wins” 
the race, and supplies the response. As the figure 
shows, not only is a choice determined from this 
process, but also the time to reach threshold. From 
trial to trial, responses and termination times will 
vary, and over trials the model will provide response 
probabilities and response time distributions that 
can be compared to a player’s data. By adjusting 
the internal model parameters in order to fit the 
observed responses, the model tells a tale about 
the player’s attention focus, memory, biases, and 
the confusability of the response alternatives. 

This decision-field theoretic model could potentially 
answer many interesting questions. For example, 
how does action orientation predict game play? 
How does decision strategy change as a result of 
learning throughout the game? Which dimensions 
receive the most attention, and which the least? 

5. DISCUSSION 

We’ve argued in this paper that cognitive modeling 
provides a detailed and dynamic view into cognition 
- at the individual player level - as it unfolds during 
video game play. Currently, this powerful approach 
is seldom utilized in game research. This is a 
shame, since cognitive modeling is a mature field, 
and there are many useful models available that 
have been affirmed by decades of research in 
carefully controlled experiments. 

Models can offer clues into the inputs required to 
produce the outputs desired. If a game is to have 
educational value, (or for communication or even 
just for fun), then variables that influence model 
behavior should be manipulated to moderate player 
behavior. Currently, most game design is guided 
by heuristics, prior experience, and flashes of 
insight. 

Many of the cognitive models in existence today are 
ready for extension to new areas. In fact, the field 
of cognitive psychology is increasingly marked by 
attempts to extend the reach of cognitive theories to 
real-world scenarios. The application of these tools 
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is especially timely considering the recent explosion 
of research into serious games (games designed to 
communicate and educate players). Designers of 
such games would likely benefit from a tool that can 
help foster a deeper understanding of what players 
focus on and are affected by during game play. 

Finally, cognitive modeling dovetails well with the 
imaging and psychophysiological research 
mentioned above. The relatively recent emergence 
of the field of cognitive neuroscience attests to this. 
Cognitive models have become so powerful that 
competition between theories is often difficult to 
assess on the basis of behavioral data alone. 
Neuroscience data is now routinely used to place 
biological plausibility constraints on computational 
models. In turn, cognitive modeling imparts a deep 
level of meaning to neuroscience results. Models 
help neuroscientists understand the cognitive 
implications of their data. 

In conclusion, cognitive modeling presents a 
powerful method for understanding what a player is 
thinking about while playing a video game. 
Research papers that describe cognitive models 
often report their model derivations in detail so that 
interested readers can adopt these methods. Our 
hope is that we’ve been able to convince readers of 
the allure of cognitive modeling for their own game 
research. 
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